Remote instruction system and method thereof

ABSTRACT

A remote instruction system includes a controller that controls an image capturing portion to capture an image of a target and transmit a captured image to a remote terminal, and controls a projecting portion to project an annotation image onto the target according to an instruction on the captured image given by the remote terminal, and a relative position changing portion that is controlled by the instruction on the captured image given by the remote terminal and changes the position of the target relative to the image projection portion and the image capturing portion.

BACKGROUND OF THE INVENTION

1. Field of the Invention

This invention relates to a remote instruction system that aninstruction can be given remotely by means of an image.

2. Description of the Related Art

In a remote repair system, remote maintenance system, remote medicalcare system, remote conferencing, and the like, for example, variousinstructions in operation procedure need to be given to a real thingfrom a remotely located position. As one of remote instruction systemswhere the instructions can be given to the real thing from the remotelocation, for example, U.S. Patent Application Publication No.2004/0070674 describes a technique of sending a moving image to a remoteterminal while a subject or target existent on the real thing side isbeing captured by a camcorder, and projecting an annotation image ontothe subject by means of a projector on the real thing side, theannotation image being designated on the basis of the captured image onthe remote terminal.

In the above-described remote instruction system, however, when aprojection position of an annotation image onto a subject is changedaccording to an instruction given by a remote terminal, a range in whichthe projection position can be changed is limited to one being displayedon a display screen of the remote terminal, namely, a range which iscaptured by a camcorder. Therefore, if the annotation image is to beprojected onto outside the afore-described range, it is necessary togive an instruction verbally to the operator or viewer who are existenton the subject side to change the position of the subject. If theposition of the subject relative to the camcorder is changed, theprojection position onto the subject projected by the projector has tobe changed accordingly. At this time, three-dimensional positionalinformation of the subject relative to the projector is necessary.However, it is not easy to obtain the three-dimensional positionalinformation of the whole subject accurately. Accordingly, it isdifficult to change the projection position of the annotation imageprecisely.

SUMMARY OF THE INVENTION

The present invention has been made in view of the above circumstancesand provides a remote instruction system, in which it is possible toenlarge a range where an annotation image can be projected onto a targetaccording to an instruction given by a remote site and improve theaccuracy of a projection position of the annotation image.

According to one aspect of the present invention, there is provided Aremote instruction system including a controller that controls an imagecapturing portion to capture an image of a target and transmit acaptured image to a remote terminal, and controls a projecting portionto project an annotation image onto the target according to aninstruction on the captured image given by the remote terminal; and arelative position changing portion that is controlled by the instructionon the captured image given by the remote terminal and changes theposition of the target relative to the image projection portion and theimage capturing portion.

According to another aspect of the present invention, there is provideda remote instruction method including: controlling an image capturingportion to capture an image of a target and transmit a captured image toa remote terminal; controlling a projecting portion to project anannotation image onto the target according to an instruction on thecaptured image given by the remote terminal; and changing a position ofthe target relative to the image projection portion and the imagecapturing portion.

With the afore-mentioned configuration, the position of the target (theviewpoint relative to the target in a remote site) can be changed fromthe remote site, thus enabling the range in which the annotation imagecan be projected to be enlarged.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the present invention will be described in detail basedon the following figures, wherein:

FIG. 1 is a block diagram showing an embodiment in accordance with aremote instruction system of the present invention;

FIG. 2 is a flowchart showing an example of process on the server in atarget end apparatus;

FIG. 3 is a flowchart showing an example of process in control signalprocessing in the flowchart shown in FIG. 2;

FIG. 4 is a flowchart showing an example of process in forming an imageon a computer of a remote terminal;

FIGS. 5A through 5C are views illustrating a change procedure of animage projection position in accordance with the rotation of a target;and

FIG. 6 is a view illustrating an example of operation in the remoteterminal.

DESCRIPTION OF THE EMBODIMENTS

A description will now be given, with reference to the accompanyingdrawings, of embodiments of the present invention.

FIG. 1 is a block diagram showing an embodiment in accordance with aremote instruction system of the present invention. In the remoteinstruction system, there are provided a target end apparatus 10 and aremote terminal 100, as shown in FIG. 1. The target end apparatus 10 andthe remote terminal 100 are connected via a network 300 so that they cancommunicate with each other. Only one remote terminal 100 is shown inFIG. 1, yet multiple remote terminals 100 can be connected to a server50 on the target end apparatus 10, as will be described later, via thenetwork 300.

Hereinafter, a target, subject, or object denotes anything to be orbeing captured by an image capturing apparatus or image capturingportion.

The target end apparatus 10 includes a first camcorder 20 that serves asa first image capturing portion, a second camcorder 30 that serves as asecond image capturing portion, a projector 40 that serves as aprojection portion, the server 50 that serves as a controller, a halfmirror 60, and a rotation stage 70 that serves as a relative positionchanging portion.

The first camcorder 20 and the second camcorder 30 are composed of, forexample, a CCD camera, and are provided in such a manner that the imageof a target 200 arranged on the rotation stage 70 can be captured fromdifferent positions. Such captured images are imported into the server50. Here, the first camcorder 20 captures the image through the halfmirror 60.

The projector 40 is composed of a crystal liquid projector or the like,and is provided so that an optical system thereof has an opticalprincipal point, which is substantially aligned with that of the firstcamcorder 20 by using the half mirror 60. The projector 40 projects anannotation image transmitted from the server 50 via the optical systemthereof and the half mirror 60. The annotation image transmitted fromthe projector 40 is reflected by the half mirror 60 and projected ontothe target 200. The annotation image may include any type of image suchas line, character, drawing, and the like.

The rotation stage 70 is controlled by the server 50, and the positionsof the target 200 relative to the first camcorder 20, the secondcamcorder 30, and the projector 40 are changed by rotating the target200 provided thereon.

The server 50 controls operations of the first camcorder 20, the secondcamcorder 30, the projector 40, and the rotation stage 70, and inaddition, sends and receives various types of information to and fromthe remote terminal 100 via the network 300. The server 50 transmits thecaptured images captured by the first camcorder 20 and the secondcamcorder 30 to the remote terminal 100, and projects the annotationimage onto the target 200, according to an instruction on the capturedimage given by the remote terminal 100. Further, the server 50 controlsto change the projection position of the annotation image projected bythe projector 40 onto the target 200, in accordance with the rotation(movement) of the target 200 together with the rotation stage 70. Atthis time, the server 50 calculates three-dimensional coordinates of theprojection position of the annotation image projected onto the target200 on the basis of the captured images captured by the first camcorder20 and the second camcorder 30, and changes the position of theannotation image projected by the projector 40 on the basis of suchcalculated three-dimensional coordinates. Specific processes will bedescribed later in detail.

The remote terminal 100 includes a display device 110 such as a crystalliquid device, CRT, or the like, a computer 120 connected to the network300, and a pointing device (mouse) 130 connected to the computer 120.The display device 110 displays the image being transmitted from thetarget end apparatus 10, on the display screen thereof. The pointingdevice (mouse) 130 is used for forming an instruction on the annotationimage to be projected onto the target 200, by manipulating variousbuttons or the like with a pointer on the display screen on which thecaptured image is being displayed, and is also used for giving aninstruction on the rotation of the target 200 by rotating the rotationstage 70.

Next, a description will be given, with reference to FIG. 2 through FIG.6, of operations of the remote instruction system having theabove-described configuration. FIG. 2 is a flowchart showing an exampleof process on the server in the target end apparatus. FIG. 3 is aflowchart showing an example of process in the control signal processing(step ST5) in the flowchart shown in FIG. 2. FIG. 4 is a flowchartshowing an example of process in creating an image on the computer ofthe remote terminal. FIGS. 5A through 5C are views illustrating a changeprocedure of the image projection position in accordance with therotation of the target. FIG. 6 is a view illustrating an example ofoperation in the remote terminal.

Referring to FIG. 2, the server 50 in the target end apparatus 10 startsimporting the captured images captured by the first camcorder 20 and thesecond camcorder 30 (step ST1), and determines whether there is aconnection request from the computer 120 of the remote terminal 100(step ST2). If there is a connection request from the computer 120, thecaptured image captured by the first camcorder 20 is transmitted to thecomputer 120 of the remote terminal via the network 300 (step ST3).

Then, the server 50 determines whether a control signal has beentransmitted from the computer 120 (step ST4). The control signal servesas a draw signal that includes information on the drawing of theannotation image and also serves as a position control signal to rotatethe rotation stage 70. If the server 50 has received the afore-mentionedcontrol signal, the server 50 implements a control signal processingaccording to the content thereof, as will be described later (step ST5).

In contrast, if the server 50 has not received the afore-mentionedcontrol signal, the server 50 determines whether there is adisconnection request issued from the computer 120 (step ST6). If thereis no disconnection request, processing goes back to step ST3 to send anewly captured image to the computer 120 of the remote terminal 100 viathe network 300. If there is a disconnection request, the server 50stops sending the captured image captured by the first camcorder 20(step ST7). Then, the sever 50 determines whether there is an endrequest of processing (step ST8). If there is no end request, processinggoes back to step ST2 to repeat the above-described processes. If thereis an end request, the sever 50 completes the process.

A description will now be given of a control signal processing in stepST5. Referring to FIG. 3, first, the server 50 determines whether thedraw signal has been received from the remote terminal 100 (step ST20).Upon receiving the draw signal, the server 50 renders an annotationimage DR on the projector 40 according to the draw signal, and projectsthe image onto the target 200 (step ST21), as shown in FIG. 5A, forexample. A portion on which the annotation image DR is projected ontothe target 200 by the projector 40 is one that the operator of theremote terminal 100 likes to indicate or annotate to the operator of thetarget end apparatus 10. Therefore, this portion is brighter than anyother portion of the target 200.

Then, the server 50 projects the annotation image DR onto the target 200with the projector 40, and calculates three-dimensional coordinates ofthe projection position of the annotation image DR on the target 200, onthe basis of the captured images captured by the first camcorder 20 andthe second camcorder 30 and the principle of triangular surveying (stepST22). As described heretofore, the projection portion of the annotationimage DR is particularly brighter than the surrounding thereof on thetarget 200. This reduces the possibility of generating miss calculationto be less than the general triangular surveying in implementing theblock matching or matching with the characteristic edge. Furthermore,the first camcorder 20 has the principal point substantially alignedwith that of the optical system of the projector 40. Hence, theprojection coordinates of the projector 40 correspond one-to-one tothose of the first camcorder 20, in spite of the shape of the target.This eliminates the necessity of searching the projection image. It istherefore possible to significantly reduce the calculation amountnecessary for calculating a distance to the target 200 to specify aposition of the three-dimensional coordinates on the surface of thetarget 200.

If the server 50 has not received the draw signal, in other words, ifthe server 50 has received the position control signal to rotate therotation stage 70, the rotation stage 70 is rotated according to arotation amount and a rotation direction according to the content of theposition control signal, as shown in FIG 5B (step ST23). If the rotationstage 70 is rotated, the projection position of the annotation image DRbeing projected onto the target 200 is changed. For this reason, first,the three-dimensional coordinates of the annotation image DR on thecaptured image captured by the first camcorder is calculated accordingto the rotation amount obtained with the above-mentioned positioncontrol signal, when the annotation image DR is moved in accordance withthe rotation of the rotation stage 70 (step ST24). Then, thethree-dimensional coordinates of the annotation image DR on theprojector 40 are calculated when the annotation image DR is movedaccording to the rotation of the rotation stage 70 (step ST26). Next,the annotation image DR subsequent to the movement is projected onto thetarget 200, as shown in FIG. 5C (step ST26).

A description will now be given of a calculation method of thethree-dimensional coordinates of the projection position of theannotation image DR, when the annotation image DR is moved (subsequentto the rotation of the rotation stage 70) in accordance with therotation of the rotation stage 70. First, three-dimensional coordinatesP0 subsequent to the rotation is obtained with the following expression,where P1 denotes three-dimensional coordinates of the projectionposition of the rotation stage prior to the rotation, which is obtainedin step ST22.{overscore (P)}0={overscore (P)}+R({overscore (P)}1−{overscore(P)}t)  [Expression 1]

However, it is supposed that a normalized direction vector n and apassing point Pt, which are represented as column vectors in thecoordinate system of the first camcorder 20. $\begin{matrix}{{n = \begin{bmatrix}{nx} \\{ny} \\{nz}\end{bmatrix}}{{Pt} = \begin{bmatrix}{Ptx} \\{Pty} \\{Ptz}\end{bmatrix}}} & \left\lbrack {{Expression}\quad 2} \right\rbrack\end{matrix}$

Here, R denotes a matrix that passes the origin and rotates around thenormalized direction vector at an angle of θ, and may be described asthe following expression. R is obtainable by the rotation amount of theposition control signal. $\begin{matrix}{R = \begin{bmatrix}{{n_{x}^{2}\left( {1 - c} \right)} + c} & {{n_{x}{n_{y}\left( {1 - c} \right)}} - {n_{z}s}} & {{n_{x}{n_{z}\left( {1 - c} \right)}} - {n_{y}s}} \\{{n_{y}{n_{x}\left( {1 - c} \right)}} + {n_{z}s}} & {{n_{y}^{2}\left( {1 - c} \right)} + c} & {{n_{y}{n_{z}\left( {1 - c} \right)}} - {n_{x}s}} \\{{n_{x}{n_{z}\left( {1 - c} \right)}} - {n_{y}s}} & {{n_{y}{n_{z}\left( {1 - c} \right)}} + {n_{x}s}} & {{n_{z}^{2}\left( {1 - c} \right)} + c}\end{bmatrix}} & \left\lbrack {{Expression}\quad 3} \right\rbrack\end{matrix}$

where c=cos θ and s=sin θ.

With respect to the afore-mentioned three-dimensional coordinates, theperspective transformation is performed by the first camcorder 20, andit is possible to calculate the coordinates of the captured imagecaptured by the first camcorder 20. As described above, the coordinatesof the captured image captured by the first camcorder 20 correspondone-to-one to the coordinates of the projection position projected bythe projector 40. Therefore, it is possible to change the position ofthe image DR according to the rotation amount of the rotation stage 70,if the image DR is projected on the basis of the afore-describedcorrespondence.

Now, the operation of the remote terminal 100 will be described. First,referring to FIG. 4, the computer 120 of the remote terminal 100 issuesa connection request to the server 50 (step ST31). Next, when theconnection is established, the computer 120, for example, displays thecaptured image being transmitted from the server 50 in the target endapparatus on the display screen of the display device 110, as shown inFIG. 6 (step ST32).

Then, the computer 120 determines whether there is an instruction, whichhas been given by the operator, on a noteworthy region in the capturedimage being displayed on a display screen 111 of the display device 110(step ST33). If there is an instruction on the noteworthy image, aprocess according to the instruction is implemented (step ST36).Specifically, if the operator of the remote terminal 100 likes toproject the annotation image onto a region in the image being displayedwhile watching the image being displayed on the display screen 111 ofthe display device 110 as shown in FIG. 6, the operator manipulates thepointing device 130 and moves a pointer Pt being displayed on thedisplay screen 111 to indicate or annotate the noteworthy region. Thenoteworthy region is a region that specifies the projection positiononto which the annotation image is projected. At this time, theinformation on the annotation image to be projected onto the noteworthyregion is indicated simultaneously. For example, by manipulating variousbuttons BT or the like provided on the display screen 111 with thepointing device 130, indicated are graphic information such as rectangleor circle to be drawn, bitmap image that has been prepared in advance,text information that has been input on the keyboard, and a pointer thatreflects the movement of the pointing device 130. The computer 120transmits the various kinds of information specified in step ST36 asdraw information to the server 50 (step ST37).

Next, the computer 120 determines whether the instruction given by theoperator of the remote terminal 100 based on the captured image beingcaptured by the first camcorder 20 has been completed (step ST38). Ifthe instruction has been completed, the computer 120 issues adisconnection request to the server 50 (step ST39), and completes theprocess. If the instruction by the operator of the remote terminal 100has not been completed, processing goes back to step ST32 to repeat theabove-described processes.

At this time, if the operator of the remote terminal 100 likes to rotatethe target 200, the operator manipulates rotation buttons R1 and R2 torotate the rotation stage 70 provided on the display screen 111 and giveinstructions on the rotation direction and the rotation amount of therotation stage 70, so as to display the region onto which the annotationimage is to be projected on the target 200 in the displayed image or soas to provide a most suitable viewpoint relative to the target 200,while watching the image being displayed on the display screen 111 asshown in FIG. 6. If there is no instruction on the noteworthy region (noinstruction on the annotation image is given) in step ST33, the computer120 determines that the operator of the remote terminal 100 gives aninstruction on the rotation of the target 200, and implements theprocess relating to the rotation of the target 200 (step ST34). Then,the computer 120 transmits the information on the rotation amount (theinformation on the movement amount) or the like to the server 50 in thetarget end apparatus (step ST35). This rotates the rotation stage 70,changes the rotation position of the target 200, and displays a newlycaptured image on the display device 110 of the remote terminal 100(step ST32).

As described heretofore, in accordance with the present embodiment, itis possible to change the position of the target 200 (the viewpointrelative to the target 200) according to the instruction given by theremote terminal 100 on the basis of the captured image, and accordinglyit is possible to enhance the instruction operability of the remoteterminal 100 and enlarge the range in which the annotation image can beprojected onto the target 200. In addition, in accordance with thepresent embodiment, even if the position of the target 200 is changedfrom the remote terminal 100, the projection position of the annotationimage is automatically changed in accordance with the afore-mentionedchange, thereby enabling the operability to be further improved.Furthermore, it is possible to precisely detect the three-dimensionalcoordinates of the projection position of the annotation image on thetarget 200, according to the principle of the triangular surveying, onthe basis of the captured images captured by the first and secondcamcorders 20 and 30. This allows the projection position of theannotation image to be adjusted accurately according to the rotation ormovement of the target 200.

In the above-described embodiment, the description has been given of theprojector exemplified as a projection portion. However, the projectionportion is not limited to the projector. For example, the image may becreated by irradiating the beam such as a laser beam onto the target.

In the above-mentioned embodiment, the rotation stage has beenexemplified as a relative position changing portion. However, therelative position changing portion is not limited to the rotation stage.For instance, a robot or the like may be applicable to the relativeposition changing portion. In the above-mentioned embodiment, thedescription has been given to the target to be moved. However, forexample, the camcorder that serves as the image capturing portion or theprojector that serves as the projection portion may be moved.

In the above-mentioned embodiment, the description has been given to therotation buttons R1 and R2 provided on the display screen 111 so as togive an instruction to rotate the rotation stage 70. However, instead ofthe rotation buttons R1 and R2, a keyboard may be utilized for giving aninstruction to rotate the rotation stage 70. An alternate method may beemployed.

In the above-mentioned embodiment, the remote terminal 100 is connectedto the server 50 via the network 300. However, an alternate method maybe employed to connect the remote terminal 100 to the server 50. Theremote terminal 100 may be provided in the target end apparatus.

In the above-mentioned embodiment, the description has been given of acase where only the captured image being captured by the first camcorder20 is transmitted to the remote terminal 100 from the server 50.However, the captured image being captured by the second camcorder 30may be transmitted to the remote terminal 100.

In the above-mentioned system, the relative position changing portionmay include a rotation stage that can rotate the target. With theafore-mentioned configuration, it is possible to project the annotationimage on the entire surrounding of the target.

According to the present invention, it is possible to enlarge the rangein which the annotation image can be projected onto the target accordingto the instruction given by the remote site and improve the accuracy ofthe projection position of the annotation image.

Although a few embodiments of the present invention have been shown anddescribed, it would be appreciated by those skilled in the art thatchanges may be made in these embodiments without departing from theprinciples and spirit of the invention, the scope of which is defined inthe claims and their equivalents.

The entire disclosure of Japanese Patent Application No. 2005-175830filed on Jun. 16, 2005 including specification, claims, drawings, andabstract is incorporated herein by reference in its entirety.

1. A remote instruction system comprising: a controller that controls animage capturing portion to capture an image of a target and transmit acaptured image to a remote terminal, and controls a projecting portionto project an annotation image onto the target according to aninstruction on the captured image given by the remote terminal; and arelative position changing portion that is controlled by the instructionon the captured image given by the remote terminal and changes theposition of the target relative to the image projection portion and theimage capturing portion.
 2. The remote instruction system according toclaim 1, wherein the controller changes a projection position of theannotation image relative to the target in accordance with a movement ofthe target made by the relative position changing portion.
 3. The remoteinstruction system according to claim 2, wherein the controllercalculates the projection position of the annotation image after themovement of the target, on the basis of the projection position of theannotation image obtained from the captured image prior to the movementof the target made by the relative position changing portion, and acontrol signal applied to the relative position changing portion.
 4. Theremote instruction system according to claim 3, further comprising afirst image capturing portion and a second image capturing portion tocapture images of the target from different positions, wherein thecontroller calculates three-dimensional coordinates of the projectionposition of the annotation image prior to the movement of the target, onthe basis of the -images captured by the first image capturing portionand the second image capturing portion.
 5. The remote instruction systemaccording to claim 4, wherein the projection portion projects theannotation image onto the target in an optical system that has anoptical principal point substantially aligned with that of the firstimage capturing portion.
 6. The remote instruction system according toclaim 1, wherein the instruction is given to the relative positionchanging portion according to an operation to change a relative positionto the captured image being displayed on a display screen of the remoteterminal.
 7. The remote instruction system according to claim 1, whereinthe relative position changing portion includes a rotation stage thatcan rotate the target.
 8. A remote instruction method comprising:controlling an image capturing portion to capture an image of a targetand transmit a captured image to a remote terminal; controlling aprojecting portion to project an annotation image onto the targetaccording to an instruction on the captured image given by the remoteterminal; and changing a position of the target relative to the imageprojection portion and the image capturing portion.
 9. The remoteinstruction method according to claim 8, further comprising a step ofchanging a projection position of the annotation image relative to thetarget in accordance with a movement of the target.
 10. The remoteinstruction method according to claim 9, further comprising a step ofcalculating the projection position of the annotation image after themovement of the target, on the basis of the projection position of theannotation image obtained from the captured image prior to the movementof the target, and a control signal used for changing the position ofthe target.